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Key findings 



All seven Regional Educational Laboratory (REL) Central states 
(Colorado, Kansas, Missouri, Nebraska, North Dakota, South 
Dakota, and Wyoming) have evaluation procedures for approving 
and reauthorizing teacher preparation programs. These procedures 
focus on program design and implementation. 

Six of these states were implementing or planning changes to their 
evaluation of teacher preparation programs to shift the focus to the 
performance of program graduates. 

Other changes in focus and process across REL Central states 
include developing statewide data collection tools, investing in data 
systems, and exploring new approaches for reporting evaluation 


findings. 
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Summary 


Policymakers, researchers, educators, and others have expressed concerns about the quality 
of teacher preparation, spurring efforts to reform how states evaluate teacher preparation 
programs. As a result, many states are rethinking their evaluation criteria and methods 
to better understand how program graduates are performing, with the goal of improving 
curricula, recruitment, and clinical experiences (Noell & Kowalski, 2010). This report 
describes how the seven Regional Educational Laboratory (REL) Central states (Colorado, 
Kansas, Missouri, Nebraska, North Dakota, South Dakota, and Wyoming) evaluate their 
teacher preparation programs and the changes they are making to improve their approach- 
es to evaluation. 

Like many states across the country, all seven REL Central states focus their evaluations of 
teacher preparation programs on program design and implementation. Evaluation teams, 
typically comprising state education agency staff and practicing and retired educators, 
review documents (on program curricula, field experiences, and candidate performance, for 
example) submitted by each teacher preparation program. The teams conduct onsite visits 
to supplement and validate the documentation and then make a determination about the 
quality of the program. 

Six of the seven REL Central states are implementing or planning changes to how they 
evaluate teacher preparation programs. Most changes involve paying more attention to 
the performance of program graduates, developing common data collection tools and data 
systems, and developing new ways to report evaluation data. Specifically, states are: 

• Increasing their focus on the performance of graduates to better assess the effec- 
tiveness of teacher preparation programs and guide improvement efforts (all but 
Wyoming). 

• Identifying or developing data collection tools for statewide use to facilitate program 
evaluation (Colorado, Kansas, Missouri, and Nebraska). These tools include assess- 
ments of content knowledge, work habits, and performance of preservice teacher 
candidates, as well as surveys of recent program graduates. 

• Investing in data systems that can support answers to evaluation questions. Colora- 
do is assessing the capacity of its data systems to link outcomes for K-12 students 
of recent graduates of teacher preparation programs to the programs they attended. 

• Exploring new approaches for reporting evaluation findings. Colorado and Missou- 
ri are developing annual reports of teacher preparation programs that focus on 
outcome measures for teacher candidates and program graduates. Nebraska is 
exploring new ways to report evaluation findings, such as a public “report card” 
with evaluation findings. 

This report provides information about how states evaluate teacher preparation programs 
for state and district education leaders and policymakers, teacher preparation program 
administrators and faculty, and others with an interest in the effectiveness of teacher 
preparation evaluation, especially in the seven REL Central states. It is based on a review 
of publicly available documents and interviews with state education agency staff with 
primary responsibility for oversight of teacher preparation programs. 
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Why this study? 


Teacher preparation is increasingly the focus of discussions among national and state 
policymakers and practitioners. 1 Concerns about the quality of teacher preparation pro- 
grams (see box 1 for definitions of key terms) have been raised by researchers, policymakers, 
analysts, teacher accreditation organizations, and teachers themselves, partly because of 
a new emphasis on teacher effectiveness and accountability (Alderman, Carey, Dillon, 
Miller, & Silva, 2011; Chesley & Jordan, 2012; Crowe, 2010; Ingersoll, Merrill, & May, 
2012; National Council for Accreditation of Teacher Education, 2010). 

Several national efforts promote reforming how teacher preparation programs are evaluat' 
ed. For example, in 2011, the U.S. Department of Education released the Plan for teacher 
education reform and improvement (U.S. Department of Education, 2011), which emphasiz- 
es better reporting of program outcomes for accountability purposes. Specifically, it rec- 
ommends that assessments include information on student achievement growth, teacher 
job placement and retention rates, and surveys of program graduates and their principals. 
These national efforts have led many states to rethink the way they evaluate teacher prepa- 
ration programs in order to better understand how graduates of different programs perform 
as teachers and to capture information to improve program curricula, teacher recruitment 
efforts, and clinical experiences (Noell & Kowalski, 2010). 


Box 1. Key terms 

Alternative teacher preparation program. A program that “primarily servejs] candidates who 
have subject matter knowledge and who are the teachers of record in a classroom while partic- 
ipating in their teacher preparation program” and “may be based in an institution of higher edu- 
cation (IHE) or outside an IHE” (U.S. Department of Education, 2013, p. x). Alternative programs 
serve individuals who have subject matter expertise but lack formal education coursework or 
training. For Title II reporting (see below), each state determines which teacher preparation 
programs are alternative programs. (See also traditional teacher preparation program.) 

Educator preparation program. A program or state-approved course of study that leads to an 
initial credential to serve as a school administrator, librarian, or counselor. 

Professional education unit. An entity with administrative oversight of one or more educator 
preparation programs. Most professional education units are schools, colleges, or depart- 
ments of education. 

Teaching certification or licensure. The state-issued documentation required for an individual 
to teach in a public school. Both terms are used in REL Central states. Three states (Missouri, 
Nebraska, and South Dakota) use the term “certification,” and four states (Colorado, Kansas, 
North Dakota, and Wyoming) use the term “licensure." 

Teaching endorsement. A descriptor that appears on a certification or licensure document to 
identify the areas in which an individual is qualified to teach. 

Teacher preparation program. Definitions vary. The U.S. Department of Education (2013, 
p. xiii) defines it as “a state-approved course of study, the completion of which signifies that 
an enrollee has met all of the state’s educational, or training requirements, or both, for an 
initial credential to teach in the state’s elementary, middle, or secondary schools." A teacher 
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Box 1. Key terms ( continued ) 


preparation program may be either a traditional program or an alternative route to certification 
or licensure. The term “teacher preparation program” is used variously to refer to the institu- 
tion that provides teacher preparation (for example, Missouri State University or the University 
of Nebraska); the program type offered within an institution (whether alternative, traditional, 
undergraduate, or graduate); or the particular certification or licensure track offered (such as 
secondary school math education). In this report, unless otherwise noted, “teacher prepara- 
tion program” refers to each subject- or grade-level-specific area in which a professional educa- 
tion unit offers preparation for teacher certification, licensure, or endorsement. 


Teacher preparation program approval and reauthorization. The process by which states for- 
mally approve teacher preparation programs, enabling them to operate in the state. These 
processes vary by state and tend to focus on program adherence to standards. New programs 
must receive initial approval before operation, and existing programs must successfully com- 
plete a reauthorization process every five to seven years. 

Teacher preparation program evaluation. Any activity designed to elicit information and make 
judgments about the quality or impact of a teacher preparation program. 

Title II of the Higher Education Act. Legislation that provides grants to state education agen- 
cies to support the recruitment and preparation of highly qualified teachers and principals and 
to support accountability for preparation programs. States are required to submit an annual 
report to the U.S. Department of Education that provides information about teacher prepara- 
tion programs and their performance. 

Traditional teacher preparation program. A program based at an institution of higher educa- 
tion, defined in Title II reporting as one that “generally servejs] undergraduate students who 
have no prior teaching or work experience, and leadjs] to at least a bachelor’s degree” (U.S. 
Department of Education, 2013, p. x). 


Evaluation 
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processes, and 
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program quality 
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states are 
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outcomes 


Nationwide, evaluation of teacher preparation programs involves primarily state program 
approval processes, which vary substantially. There is no systematically collected infer- 
mation about these processes (Feuer, Floden, Chudowsky, & Ahn, 2013). Participants in 
REL Central’s Educator Effectiveness Research Alliance have expressed the need for better 
information about how to evaluate these programs. Alliance participants include state edu- 
cation agency administrators, teacher preparation program administrators and faculty, and 
researchers. To begin to fill this information gap and respond to alliance member needs, 
this study examines REL Central state efforts to evaluate teacher preparation programs. 

State teacher preparation program evaluations are shifting their focus to program quality and outcomes 

All states have standards for approving and reauthorizing teacher preparation programs, 
and many also encourage or require these programs to seek accreditation from an exter- 
nal agency (Cochran-Smith & Zeichner, 2005). Evaluation standards have tended to 
emphasize the quality of program processes, and few states have engaged in evaluation that 
focused on both program quality and outcomes (Crowe, 2011). However, some states are 
changing their evaluations to focus on program outcomes. 

Based on the most recent data available, as of early 2012, at least 13 states (none served 
by REL Central) and the District of Columbia reported (or were planning to report) 
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information about K-12 student achievement (using value-added statistical models, also 
known as student growth models) in their teacher preparation program evaluations. Several 
jurisdictions also plan to use additional measures, such as indicators of program graduate 
“persistence” (the length of time a graduate remains in teaching), graduate placement in 
high-needs fields, certification or licensure exam scores, advancement to higher certifica- 
tion or licensure, beginning teaching performance, and graduate surveys (Sawchuk, 2012). 
Louisiana and Tennessee are recognized as leaders in evaluating teacher preparation pro- 
grams because they have implemented systems that use value-added student achievement 
measures to evaluate effectiveness (U.S. Department of Education, 2011). 


Reviews of research and evaluation related to teacher preparation have identified several 
quality and outcome measures used to evaluate teacher preparation programs (table 1). 
Measures of program quality focus on program design and implementation, including selec- 
tion processes, program resources, program curricula, and clinical experiences. Measures 
of program outcomes include teacher knowledge and skills, teacher perceptions, teacher 
placement and persistence, employer perceptions, teacher evaluation results, and student 
achievement outcomes. Despite an increasing focus on both process and outcome mea- 
sures, there is little consensus about the relative merits of using particular measures, data 
sources, or methods for evaluation (Floden, 2012; Noell, Brownell, Buzick, & Jones, 2014). 

Recent national initiatives are motivating states to focus on evaluating teacher preparation programs 

A focus on evaluating teacher preparation programs is reflected in several recent federal 
policy efforts. As part of proposed changes to the Title II teacher quality and accountability 
provisions of the Higher Education Act, the U.S. Department of Education (2011) promot- 
ed increased reporting of program outcomes, including student learning, satisfaction, and 
employment outcomes. In April 2014 President Barack Obama directed the department to 
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Table 1. Measures used in evaluation of teacher preparation programs, with 
examples 


Measures of program quality Measures of program outcomes 3 


Selection processes (admission 
requirements, candidate achievement 
test scores) 

Program resources (faculty 
qualifications) 

Program curriculum (emphasis on 
content, general arts and sciences, 
education foundations, education 
methods courses) 

Clinical experiences (nature and 
extent of field and student teaching 
experiences) 


Teacher knowledge and skills (scores on certification or 
licensure tests or performance assessments) 

Teacher perceptions (ratings of satisfaction with their 
preparation program and preparedness to teach) 

Teacher placement (number of graduates hired as full-time 
teachers) 

Teacher persistence (length of time graduates remain in a 
school, district, or K-12 teaching) 

Employer perceptions (principals’ ratings of teacher 
knowledge and skills) 

State and district teacher evaluation results (evaluation 
ratings based on teaching practice and outcomes) 

Student achievement (value-added estimates based on 
state achievement tests) 


a. Teacher and student data are aggregated to the program level and used to compare outcomes among pro- 
grams and with benchmarked standards. 

Note: This table summarizes a range of quality and outcome measures that have been used to evaluate 
teacher preparation programs. Categories of measures and examples were identified by the authors. 

Source: Adapted from Coggshall, Bivona, & Reschly (2012); Feuer et al. (2013); and Zeichner & Conklin 
(2005). 
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develop a plan to encourage states to develop meaningful systems for identifying high- and 
low-performing teacher preparation programs, move from input-focused measures to mean- 
ingful outcome measures, improve the availability of information about programs, and use 
state ratings to guide allocation of Teacher Education Assistance for College and Higher 
Education grant funds, which subsidize training for teachers who commit to serve in high- 
need fields (The White House, Office of the Press Secretary, 2014). 

Improved accountability for teacher preparation programs is also promoted by the U.S. 
Department of Education through its Race to the Top initiative. Grantee states are 
required to link data on student achievement and student achievement growth to teachers, 
tie this information to the in-state programs that prepare teachers, report data on program 
effectiveness publicly for each preparation program in the state, and expand teacher educa- 
tion programs and program components that produce graduates who are effective teachers 
(Crowe, 2011; U.S. Department of Education, 2009). 

Other recent national initiatives include the efforts of the Teacher Performance Assess- 
ment Consortium, a 25-state initiative of some 200 teacher preparation programs, to 
develop a teacher performance assessment that predicts effective teaching and student 
learning and informs teacher development (http://edtpa.aacte.org/); studies of teacher 
preparation programs by the National Council for Teacher Quality to distinguish more 
and less effective programs based on adherence to standards as reflected in program doc- 
umentation (for example, Greenberg, McKee, & Walsh, 2013); and the revised Council 
for the Accreditation of Educator Preparation (2013) standards, which promote the use 
of evidence for teacher preparation program accountability. A 2012 report by the Task 
Force on Educator Preparation and Entry into the Profession called for transformation in 
educator preparation (Council of Chief State School Officers, 2012); the task force includ- 
ed members of the Council of Chief State School Officers, the National Association of 
State Boards of Education, and the National Governors Association. The report encour- 
ages states to hold preparation programs accountable by establishing a performance-rating 
system to guide program improvement and state approval of teacher preparation programs. 
A review of accountability systems of state teacher preparation programs by the Center 
for American Progress concluded that they are of limited utility for identifying low-per- 
forming programs and recommended that states adopt common indicators of program and 
graduate performance as a basis for teacher preparation program evaluation (Crowe, 2010). 

What the study examined 
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The study team reviewed existing documents and conducted interviews with state edu- 
cation staff with responsibility for oversight of teacher preparation programs in each REL 
Central state 2 to examine the following research questions about the evaluation of teacher 
preparation programs: 

• How are states evaluating teacher preparation programs in REL Central states? 

• What changes are planned or under way for state evaluation of teacher prepara- 
tion programs in REL Central states? 

The methodology used in this descriptive study is described briefly in box 2 and in more 
detail in appendix A. 
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Box 2. Data and methods 


Study design. This descriptive study is based on a review of publicly available documents 
with information about current and planned approaches for evaluating teacher preparation pro- 
grams in REL Central states, as well as interviews with state education agency representatives 
with primary responsibility for oversight and approval of teacher preparation programs. 


Data collection. Document reviews and staff interviews took place between May and October 
2013. Publicly available documents produced in 2007 or later were accessed from websites, 
including those associated with state education agencies, departments of higher education, 
journals, conferences, colleges and universities, and research organizations (appendix B 
lists the documents reviewed). Interviews were conducted by telephone with one or two state 
education agency staff members in six of the seven REL Central states (a respondent from 
Wyoming could not be reached to participate in an interview). The structured interview protocol 
contained questions about current and planned approaches for evaluating teacher training 
programs (see appendix C). 

Data analysis. Analysis of the information collected through the document reviews and inter- 
views focused on six content domains that reflect key elements of state teacher preparation 
program evaluation activities: questions or objectives being served by program evaluation, 
program standards, measures, reporting conventions, state education agency use of program 
evaluation findings, and confidentiality provisions. Two study team members, working closely 
with the principal investigator, reviewed and coded the documents. The study team generated 
reports of coded text from documents and interviews for each content domain. These reports 
were used to create written summaries of the findings. To ensure accuracy, relevant draft 
report sections of the summaries were sent to each state for review. 
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Teacher preparation in REL Central states 


This section provides background information about teacher preparation in REL Central 
states and state oversight of teacher preparation programs. 

REL Central states serve more than 44,000 teacher candidates through more than 170 teacher 
preparation programs 

State Title II reports, which present descriptive information about state teacher prepa- 
ration programs to the U.S. Department of Education, show that 172 programs offered 
teacher preparation during the 2010/11 academic year in REL Central states (figure l). 3 
The region’s three most populous states (Colorado, Kansas, and Missouri) accounted for 
about threeTourths of the programs. More than two-thirds of the programs were identified 
as traditional, meaning that they typically serve undergraduate students with no teaching 
experience and lead to a bachelor’s degree. 

Across the region about 44,000 teacher candidates were enrolled in programs and nearly 
14,000 completed them during the 2010/11 academic year (figures 2 and 3). 4 Because most 
programs take more than a year to complete, the number of program completers in a given 
year is much smaller than the number enrolled in that year. While alternative programs 
accounted for nearly a third of teacher preparation programs during the 2010/11 academic 
year, 10 percent of program completers attended an alternative program. 
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Figure 1. Most REL Central state teacher preparation programs were traditional in 
2010/11 


Traditional (n = 121) ■ Alternative, IHE-based (n = 26) ■ Alternative, not IHE-based (n = 25) 


Colorado 



Wyoming 


30 40 

Number of programs 


IHE is institution of higher education. 

Note: For Title II reporting, states count multiple certification or licensure tracks at a single institution of 
higher education as a single program. Traditional programs typically serve undergraduate students with no 
teaching experience and lead to a bachelor's degree. Alternative programs, which can be based inside or 
outside an institution of higher learning, serve candidates who have subject matter knowledge but lack formal 
teacher training and who teach in a classroom while participating in the program. 

Source: Authors' summary of data reported in U.S. Department of Education (2013). 


Figure 2. REL Central state teacher preparation programs enrolled 44,000 
candidates in 2010/11 

■ Traditional (n = 40,395) ■ Alternative, IHE-based (n = 1,699) ■ Alternative, not IHE-based (n = 1,894) 
Colorado 

Kansas 
Missouri 
Nebraska 
North Dakota 
South Dakota 
Wyoming 

0 5,000 10,000 15,000 

Number of enrollee s 



IHE is institution of higher education. 

Note: For Title II reporting, states count multiple certification or licensure tracks at a single institution of 
higher education as a single program. Traditional programs typically serve undergraduate students with no 
teaching experience and lead to a bachelor's degree. Alternative programs, which can be based inside or 
outside an institution of higher learning, serve candidates who have subject matter knowledge but lack formal 
teacher training and who teach in a classroom while participating in the program. 

Source: Authors' summary of data reported in U.S. Department of Education (2013). 
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Figure 3. Nearly 14,000 candidates completed teacher preparation programs in 
REL Central states in 2010/11 


■ Traditional (n = 12,157) ■ Alternative, IHE-based (n = 639) ■ Alternative, not IHE-based (n = 910) 
Colorado 



Wyoming 


1,000 


2,000 3,000 

Number of completers 


4,000 


5,000 


IHE is institution of higher education. 

Note: For Title II reporting purposes states count multiple traditional programs at a single institution of higher 
education as a single program. Traditional programs typically serve undergraduate students with no teaching 
experience and lead to a bachelor's degree. Alternative programs, which can be based inside or outside an 
institution of higher learning, serve candidates who have subject matter knowledge but lack formal teacher 
training and who teach in a classroom while participating in the program. 

Source: Authors' summary of data reported in U.S. Department of Education (2013). 


As of October 
2013, all seven 
REL Central states 
had procedures in 
place for approving 
and reauthorizing 
all teacher 
preparation 
programs in 
the state 


In REL Central states alternative programs are concentrated in Colorado, Kansas, and 
Missouri. Programs in these states tend to serve individuals who have subject matter exper- 
tise but lack formal education coursework or training. Candidates in these programs often 
receive short-term licenses that allow them to teach under supervision while completing 
courses to meet state certification or licensure requirements. In Colorado, which has the 
highest proportion of alternative program completers among REL Central states, alterna- 
tive preparation is provided by “designated agencies,” which, in addition to institutions 
of higher education, include school districts, charter or private schools, regional boards 
of cooperative educational services, and nonprofit organizations. Alternative programs in 
Colorado also tend to serve candidates in rural settings, where there are often no qualified 
applicants for open teaching positions. 

State oversight of teacher preparation programs varies across REL Central states 

As of October 2013, all seven states had procedures in place for approving and reautho- 
rizing all teacher preparation programs in the state. 5 In four states (Missouri, Nebraska, 
North Dakota, and Wyoming) the same entity that approves and reauthorizes teacher 
training programs conducts the evaluations (table 2); in the remaining states (Colorado, 
Kansas, and South Dakota) the functions are conducted by separate entities. In five states 
(Colorado, Kansas, Missouri, Nebraska, and South Dakota) the department of education 
is charged with evaluating and authorizing teacher training programs; in North Dakota 
and Wyoming external boards are responsible for evaluating and for approving and reau- 
thorizing of teacher preparation programs. In North Dakota, for example, the Education 
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Table 2. REL Central state entities that oversee and evaluate teacher preparation 
programs, by state, as of October 2013 


State 

Entities that approve and 
reauthorize programs 

Entities that evaluate programs 

Colorado 

Colorado State Board of Education 
Colorado Commission on Higher Education 

Colorado Department of Higher Education 
Colorado Department of Education (Office of 
Professional Services and Educator Licensure) 

Kansas 

Kansas State Board of Education 

Kansas State Department of Education 
(Office of Teacher Licensure and Accreditation) 

Missouri 

Missouri Department of Elementary and Secondary Education (Office of Educator Quality) 


Nebraska 


Nebraska Department of Education 


North Dakota 


North Dakota Education Standards and Practices Board 


South Dakota South Dakota State Board of Education 


South Dakota Department of Education 
(Office of Accreditation and Teacher Quality) 


Wyoming 


Wyoming Professional Teaching Standards Board 


Source: Authors' analysis of publicly available documents and interviews with state education agency staff, as 
described in appendix A. 


Standards and Practices Board, funded by educator licensure fees, is responsible for teacher 
licensure and professional development in addition to program evaluation and approval. 
Its 10 members, appointed by the governor, include teachers, administrators, school board 
members, and teacher educators. 

Evaluation of teacher preparation programs in REL Central states 


State approval and reauthorization processes are the primary means through which teacher 
preparation programs are evaluated in REL Central states as of October 2013. These pro- 
cesses are used primarily to ensure that programs meet legislative requirements and stan- 
dards established by each state. States can deny approval or reauthorization to programs 
that fail to meet standards. Programs are evaluated for initial approval and then for ream 
thorization every five to seven years. 


All seven REL 
Central states 
focused their 
evaluations 
of teacher 
preparation 
programs on 
the program’s 
design and the 
implementation 
of that design, 
though most 
state education 
agency officials 
described plans 
for more frequent, 
outcomes-based, 
or program- 
improvement- 
oriented evaluation 
approaches 


Reviews of programs are conducted within professional education units, such as schools, 
colleges, or departments of education. Each state has its own standards for program approv- 
al and reauthorization, and three states (North Dakota, South Dakota, and Wyoming) 
explicitly require programs to meet national standards in addition to state standards, such 
as those developed by the National Council for the Accreditation of Teacher Education/ 
Council for the Accreditation of Educator Preparation (NCATE/CAEP), 6 the Interstate 
Teacher Assessment and Support Consortium, and the Interstate School Leaders Licen- 
sure Consortium. The state entities that review teacher preparation programs analyze 
program documentation (such as descriptions of program curricula, field experiences, and 
candidate performance) submitted by each professional education unit and conduct on-site 
visits to supplement and validate the information submitted. 

At the time of this study, all seven REL Central states focused their evaluations of teacher 
preparation programs on the program’s design and the implementation of that design 
(table 3), though most state education agency officials described plans for more frequent, 
outcomes-based, or program-improvement-oriented evaluation approaches (see section 
below on planned changes). 
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Table 3. Summary of state teacher preparation program evaluation in REL Central 
states, as of October 2013 

State 

Frequency 

(after initial approval) 

Data sources 

Other features 

Colorado 

Not more than once 
every five years 

Program documents, site visits 

Feedback provided to guide 
improvement; programs 
encouraged to collect and 
report teacher and student 
outcome data 

Kansas 

Every seven years 

Program documents, site visits 

Improvement process for 
programs with identified 
areas for improvement 

Missouri 

Annually beginning in 
2013/14 (formerly, 
every seven years) 

Candidate grade point average, 
candidate content knowledge 
scores, teacher and school leader 
ratings of satisfaction with program 
quality (additional data sources to 
be included in future years) 


Nebraska 

Annually (limited) and 
every seven years 
(comprehensive) 

Program documents, site visits 


North Dakota 

Every seven years 

Program documents, site visits 


South Dakota 

Every seven years 

Program documents, site visits 


Wyoming 

Every seven years 

Program documents 


Source: Authors' analysis of publicly available documents and interviews with state education agency staff, as 
described in appendix A. 


Several state officials reported that state agencies routinely collect information from 
programs about participant enrollment and completion rates. These data can be used to 
monitor the preparation of teachers who can meet needs in harchtostaff schools, subjects, 
and grade levels. Officials in Colorado and Kansas indicated that evaluations are used 
to encourage program improvement. In Colorado, program evaluation data are used to 
guide state support for program improvement. In Kansas, programs identified as needing 
improvement must submit annual progress reports to demonstrate improvement efforts. 

Colorado and North Dakota state education agency officials indicated that some districts 
and programs have been leaders in evaluation efforts focused on outcomes. A Colorado 
state agency staff member said that districts have led program evaluation efforts and that 
some districts and programs have indicated an interest in sharing data to facilitate program 
evaluation. A North Dakota state agency staff member mentioned that many programs 
collect their own data on candidate outcomes and that several programs are exploring 
greater use of tools to measure the performance of teacher candidates. 

Colorado 

Institutions offering teacher preparation programs are required to seek initial program approv- 
al from the state and reauthorization once every five years. State evaluation has focused on 
program approval and reauthorization, to assess whether programs meet state standards in stat- 
utes, state board of education rules, and state department of education policy and guidance . 7 

The evaluation process includes a review of written proposals and submitted documents 
(such as course syllabi) to assess the program’s adherence to standards and rules. Site visits 


Several state 
officials reported 
that state agencies 
routinely collect 
information 
from programs 
about participant 
enrollment and 
completion rates 
that can be used 
to monitor the 
preparation of 
teachers who 
can meet needs 
in hard-to-staff 
schools, subjects, 
and grade levels 
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are conducted for programs seeking reauthorization, and feedback is provided to guide 
program improvement. A state agency staff member described close collaboration with 
program staff to drive improvement: “[We try to work] together and not have the state 
‘do unto.’” Agencies that offer alternative programs are held to the same standards as tra- 
ditional, university-based programs, and state statutes include criteria to ensure that they 
specify goals, such as focusing on serving hard-to-staff and high-need areas. 


Some school districts and agencies that offer alternative programs have examined program 
outcomes. For example, one large urban school district examined the effectiveness of its 
teachers prepared in different programs through student achievement outcomes. “This dis- 
trict is actually using its own evaluation framework as the lead framework to compare all 
the graduates of those preparation programs in terms of the students in their classrooms’ 
achievement,” according to a state agency staff member. The staff member also described 
interest among district and teacher preparation program administrators in greater collabo' 
ration and data sharing for examining program outcomes and supporting program improve- 
ment. The state has also used the program reauthorization process to build capacity for 
documenting program outcomes over time: “We created a pilot where designated agencies 
give each other a lot more critical feedback around how [to] actually document successes 
and challenges. I think if we can do more of that work, we can increase the quality across 
the board, and we can also increase what [program outcome] measures we want to collect.” 
Although evidence of outcomes for teachers and their students is not required for ream 
thorization, the Colorado Department of Education asks programs whether they collect 
outcome data and how they use it to assess programs. 

Kansas 


Colorado requires 
institutions 
offering teacher 
preparation 
programs to seek 
reauthorization 
once every five 
years, while 
Kansas, Missouri, 
Nebraska, North 
Dakota, and South 
Dakota require 
reauthorization 
every seven years 


All institutions in Kansas that offer teacher preparation programs leading to licensure are 
required to seek initial program approval and to seek re authorization every seven years. 
The process involves site visits and examination of documents submitted by the programs 
to determine whether they meet state standards. Course syllabi are reviewed during the 
approval process for new programs. Documents submitted for reauthorization by programs 
must include assessments of progress toward meeting state licensure standards, including 
use of the Praxis Series tests for content knowledge assessment; of ability to plan instruc- 
tion; of clinical experience; of teachers’ effect on student learning; and of content knowl- 
edge through comprehensive exam scores, portfolio reviews, case studies, and course 
grades. A template, similar to that used for recognition of specialized professional associ- 
ation programs, is used to solicit information about program context (such as admission 
criteria, course requirements, and student enrollment); assessments used by the program to 
determine whether candidates meet standards, and the standards each assessment address- 
es; evidence for meeting standards; and a description of how assessment results are used to 
improve candidate and program performance. 

A program review team examines documentation submitted by programs and reports find- 
ings to the Kansas State Department of Education Evaluation Review Committee. The 
committee makes a recommendation for program approval or disapproval to the state 
board of education, which makes the final decision. Programs may be approved with areas 
identified for improvement, in which case the institution is required to submit to the state 
an annual report documenting its improvement efforts. A program with critical deficien- 
cies is designated as “approved with stipulation” and has two years to address them. 
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Missouri 


Evaluations of teacher preparation programs are conducted by the Office of Educator 
Quality, which recently developed new standards and is revising its processes to increase 
the frequency of program evaluation and the focus on program outcomes. Before these 
revisions all institutions offering teacher preparation programs were required to seek initial 
approval and then to seek reauthorization every seven years. Information was collected 
through document reviews and site visits and included information about admissions 
criteria, faculty and resources, field and clinical experiences, and teacher candidate per- 
formance. An annual evaluation process began at the end of the 2013/14 academic year, 
based on three measures: candidate grade point average, content knowledge (Praxis Series 
test) scores, and ratings from beginning teacher and school leader surveys that measure 
satisfaction with the quality of teacher preparation. 

Nebraska 

All institutions offering teacher preparation programs are required to receive initial 
program approval, an annual review, and a comprehensive seven-year review. Programs 
must meet requirements outlined in the Nebraska Administrative Code, known as Rule 
20. 8 For the seven-year review, institutions are required to prepare documentation for each 
certification area, including syllabi and other program documents. They are also required 
to prepare documentation of candidate performance in six to eight assessment areas, 
including grade point average, employer follow-ups, work samples, and other data used to 
measure candidate performance. Institutions may also provide information about program 
improvement, as needed. The seven-year review includes an on-site visit by a review team 
that conducts interviews to substantiate information in the materials submitted by the 
programs. The Nebraska Department of Education decides program approvals and ream 
thorizations based on a review of the documents and information collected during the 
on-site visit. 


In 2014 evaluation 
in Missouri began 
to occur annually 
and to be based on 
candidate grade 
point average, 
candidate content 
knowledge, and 
beginning teacher 
and school 
leader ratings 
of satisfaction 
with the quality 
of teacher 
preparation 


North Dakota 

All programs that lead to licensure, endorsements, or education degrees are required to 
submit reports to the Educational Standards and Practices Board and host an on-site visit 
to seek initial approval and then reauthorization at least once every seven years. The 
professional education unit within each institution is expected to meet NCATE/CAEP 
standards as well as North Dakota Program Approval Standards for individual programs. 
North Dakota has also adopted Interstate Teacher Assessment and Support Consortium 
standards for teacher preparation programs and Interstate School Leaders Licensure Con- 
sortium standards for leader preparation programs. Program approval standards and pro- 
cedures are reviewed every five years, at a minimum, with input from representatives of 
teacher preparation programs and K-12 educators. 

Programs are required to submit documentation, such as program and course cata- 
logs, policy documents, and descriptions of curricula. Programs must demonstrate that 
80 percent of their graduates achieve passing scores on the Praxis Series tests, which are 
administered as part of the licensure process. Course syllabi and student work are reviewed 
during on-site visits. Professional education units also have the option of pursuing NCATE/ 
CAEP accreditation; half of the state’s programs are accredited. 
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Many programs collect their own data on teacher outcomes. A state agency staff member 
said: “North Dakota is a very local control state. Much [program evaluation] is done at 
the institutional level. We don’t require or collect it at the state level.” The staff member 
also indicated that some programs are considering adopting measures for candidates that 
examine classroom teaching performance, and the state expects to consider establishing a 
common statewide measure. 


South Dakota 


All institutions offering teacher preparation programs are required to seek initial program 
approval through the state board of education and then reauthorization every seven years. 
In addition to state board of education approval, professional education units offering 
teacher preparation programs must also be accredited by NCATE/CAEP or by a regional 
accrediting agency. State program reviews evaluate adherence to state administrative rules, 
which require that professional education units use data to evaluate the efficacy of their 
program components, including field and clinical experiences. Programs are also expected 
to share evaluation data with teacher candidates and faculty to help them reflect on and 
improve their performance. 

Programs submit documents to the state as part of an institutional self-study report, includ- 
ing information about program conceptual frameworks, program assessments, admission 
criteria, course requirements, and field experiences. The state program review also includes 
examination of data from at least six assessments related to teacher candidate performance. 
Programs must use Praxis Series tests to assess teacher candidates’ content knowledge, but 
they have discretion regarding other assessments. Site visits by a review team verify the 
information submitted. Following the review, the South Dakota Department of Educa- 
tion makes a program approval recommendation to the state board of education. Programs 
that do not receive approval are placed on probation and required to address identified 
deficiencies. 


Teacher 
preparation 
programs in 
South Dakota 
are expected to 
share evaluation 
data with teacher 
candidates and 
faculty to help 
them reflect on 
and improve their 
performance 


Wyoming 

To be approved by Wyoming’s Professional Teaching Standards Board, teacher prepa- 
ration programs must be accredited by NCATE/CAEP or successfully pass the board’s 
review process, which is aligned with NCATE/CAEP’s. All teachers prepared in Wyoming 
complete programs at the University of Wyoming. Programs are reviewed for reauthori- 
zation by the board every seven years. In addition to meeting NCATE/CAEP standards, 
programs must also meet specialized professional association standards, such as those 
established by the Council for Exceptional Children, the National Council for Teach- 
ers of English, and the National Council for Teachers of Math. Programs in disciplines 
without an associated specialized professional association must adhere to the state’s stan- 
dards, as outlined in Wyoming Rules and Regulations Governing Licensing for School 
Personnel. NCATE’s state evaluation protocol indicates that professional education units 
must provide data for each of their programs from assessments administered at the time of 
candidate admission, state licensure tests, field or internship assessments, and candidate 
follow-up assessments. 
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Planned changes in evaluating teacher 
preparation programs in REL Central states 


As of October 2013, six of the seven REL Central states are implementing or planning 
changes to how they evaluate teacher preparation programs. States are moving from a 
focus on program approval and reauthorization processes designed to ensure compliance 
with standards to more frequent evaluations that emphasize program outcomes for teacher 
candidates, practicing teachers, and their students. To enable this switch in focus, states are 
developing and implementing new data collection tools and enhancing their data systems. 
States are also exploring ways to report evaluation results that support public accountabili- 
ty for outcomes and program improvement. 


Plans for changes to program evaluations in REL Central states are consistent with the 
growing federal emphasis on outcome-focused evaluation. Through proposed changes to 
Title II regulations included in the Higher Education Act and the Race to the Top initia- 
tive, the U.S. Department of Education has encouraged states to focus on student learn- 
ing outcomes, teacher employment outcomes, and teacher and school leader satisfaction 
with program quality. State plans for program evaluations address these outcome areas and 
include others such as assessments of teachers’ classroom practice, knowledge, and skills. 

State efforts vary, but most include increased attention to the performance of program 
graduates, development of common data collection tools and data systems, and develop- 
ment of new approaches for reporting evaluation data (table 4). 


States are 
exploring ways to 
report evaluation 
results that 
support public 
accountability 
for outcomes 
and program 
improvement 


Focus on the performance of program graduates 

At least six REL Central states are implementing or planning changes to state evaluation 
of teacher preparation programs to focus on the performance of program graduates. That 
focus is intended to provide additional information to judge program effectiveness. In at 
least two states, graduate outcome data are intended to guide program improvement efforts. 

Colorado. State policy changes have required modifying Colorado’s objectives for program 
evaluation — from ensuring compliance with state standards to measuring the effectiveness 
of programs as evidenced by the performance of program graduates. State laws passed in 
2010 and 2011 increased the requirements for program evaluation and reporting, with an 
emphasis on assessing the performance of program graduates. 9 

As of late 2013, state law requires annual reports to the General Assembly that include infor- 
mation about the performance of the state’s educator preparation programs in terms of student 
academic growth, educator placement, and educator mobility and retention. The Colorado 
Department of Education intends to use data about the performance of graduates and their 
students to identify which educator preparation programs produce the most effective graduates 
(in particular subject areas, with particular types of students, and in particular academic set- 
tings) and to better understand what makes programs effective. Colorado has developed a new 
system to evaluate educators, which began implementation during the 2013/14 academic year. 
Because the Colorado Teacher Quality Standards used to evaluate teachers differ from stan- 
dards in state statutes for evaluating educator preparation programs, the Colorado Department 
of Education has been working with leaders of teacher training programs to align the two sets 
of standards so that teachers can be prepared to meet the expectations of future evaluations. 
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Table 4. Changes planned for state teacher preparation program evaluation in REL Central states 

State 

Focus on performance of 
program graduates 

Identification and 
development of common 
data collection tools 

Enhancements to 
state data systems 

New approaches 
for reporting 
evaluation findings 

Colorado 

Revising evaluations to focus on 
student academic growth, educator 
placement, and educator mobility 
and retention 

Developing a graduate 
survey 

Investing in state agency 
data systems to support 
new approaches to 
evaluation 

Developing annual 
program performance 
reports that will be 
publicly available 

Kansas 

Planning to evaluate programs 
on K-12 student performance 
and exploring ways to incorporate 
additional teacher performance data 

Examining ways to 
compare ratings 
from multiple teacher 
performance measures 



Missouri 

Developing an annual system for 
evaluating programs that initially 
focuses on candidate grade point 
average and knowledge test scores 

Requiring new 
assessments for 
candidates, focusing on 
knowledge, performance, 
and work habits 


Developing annual 
program performance 
reports that will be 
publicly available 

Nebraska 

Planning to incorporate program 
outcome data in its evaluation 
efforts, including measures aligned 
with revised teacher performance 
standards 

Developing a graduate 
follow-up survey 
and tools to assess 
candidate knowledge 
and performance 


Considering program 
evaluation reports that 
will be publicly available 

North Dakota 

Considering the use of K-12 student 
achievement and teacher evaluation 
data for program evaluation 




South Dakota 

Considering the use of teacher 
performance data for program 
evaluation 




Wyoming 

A respondent from Wyoming could 
not be reached to participate in 
an interview, and a review of state 
documents identified no planned 
changes for teacher preparation 
program evaluation 




Source: Authors' analysis of publicly available documents and interviews with state education agency staff, as described in appendix A. 


Kansas. Although the evaluation of teacher preparation programs does not currently con- 
sider teacher performance, Kansas is developing a system in which K-12 student perfon 
mance will become part of a state evaluation system. The Kansas State Department of 
Education collects work sample data from teacher candidates. About half the state’s prepa- 
ration programs use the Kansas Performance Teaching Portfolio to demonstrate candi- 
dates’ ability to design and implement a unit of study. The remaining programs use similar 
tools. The state is exploring ways to ensure that teacher candidate performance data are 
comparable across teacher preparation programs. 

Missouri. Missouri is using new state standards for educator preparation to develop a 
system to annually evaluate and report on the performance of teacher preparation pro- 
grams. According to a state agency staff member, the state is moving away from a doc- 
ument-based review every seven years to an indicator-based system that tracks annual 
performance: “[We have] continued to look at those same data points except [we now] 
capture them annually and eliminate all the other text that went with it.” The Missouri 
Standards for the Preparation of Educators are based on six program areas: academics, 
design and assessment, field and clinical experiences, candidates, faculty, and operations 
and resources. 
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Initial implementation of the new annual evaluation system focuses on three measures: 
candidate grade point average, candidate knowledge (Praxis Series test scores), and results 
from beginning teacher and school leader surveys. According to state agency staff, the 
state intends to develop annual indicators that align with the new standards, and programs 
will be expected to meet annual benchmarks. If a program fails to meet benchmarks for 
two years and shows no improvement over that two-year period, Missouri’s Office of Edu- 
cator Quality will work with the program to develop an improvement plan to be approved 
by the state board. If the program fails to demonstrate improvement for an additional two 
years, it will be subject to discontinuation. Depending on the circumstances and program 
structure, a professional education unit or programs in certain certification areas could lose 
authorization. 

Nebraska. The state is changing its process for evaluating teacher preparation pro- 
grams from an annual evaluation focused on monitoring and compliance to one focused 
on program outcomes and continuous improvement. According to a state agency staff 
member, the state plans to incorporate revised Interstate Teacher Assessment and Support 
Consortium standards, developed by the Council of Chief State School Officers (2012), 
into state regulations for teacher preparation programs. These standards focus on K-12 
learner development and differences, learning environments, content knowledge, applica- 
tion of content, assessment, planning for instruction, instructional strategies, profession- 
al learning and ethical practice, and leadership and collaboration. A state agency staff 
member reported that incorporating these standards into state regulations will likely lead 
to expanded state requirements for program evaluation, including demands for additional 
data: “We have, in my opinion, a fairly input-driven approval process, [and] we are trying 
to move that into a more outcome and data-driven process.” 

North Dakota. Teacher preparation programs in North Dakota are participating in a 
multistate initiative to compare student achievement outcomes for graduates. The state 
hopes to use the results of these comparisons in revising its evaluation methods. The ini- 
tiative is led by the Bush Foundation in partnership with North Dakota State Universi- 
ty and Valley City State University. A state agency staff member indicated that North 
Dakota is revising how teachers and principals are evaluated (for example, it is consider- 
ing collecting data on teacher classroom practices), which may in turn affect how teacher 
preparation programs are evaluated. For example, the state is considering asking schools 
to generate school composite scores based on aggregated data from evaluations of teachers 
and principals. Concerns about misuse of those data (for example, for ranking schools) are 
a central part of that conversation. 

South Dakota. State agency staff mentioned that, as the statewide longitudinal data 
system is developed, South Dakota will begin to track performance outcomes for teacher 
candidates and licensed teachers and that these data will be linked to teacher preparation 
programs to evaluate program effectiveness. 

Identification and development of common data collection tools 

Four REL Central states (Colorado, Kansas, Missouri, and Nebraska) are identifying or 
developing common data collection tools to facilitate evaluation of teacher preparation 
programs. These tools include assessments of content knowledge, work habits, and perfor- 
mance for teacher candidates, as well as surveys of recent program graduates. 


Four REL 
Central states 
are identifying 
or developing 
common data 
collection tools 
to facilitate 
evaluation 
of teacher 
preparation 
programs 
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Colorado. A state agency staff member reported that the deans of teacher preparation 
programs have been developing a common survey to be administered to graduates of all 
institutions offering teacher preparation programs to assess perceptions of program quality. 

Kansas. The state is updating the Kansas Performance Teaching Portfolio to reflect the 
updated Interstate Teacher Assessment and Support Consortium standards. Ratings coh 
lected through the portfolio will be compared with ratings from similar tools used by pro- 
grams to assess the utility of these tools for program evaluation. 


Missouri. To better reflect the state’s new standards for teacher preparation, Missouri is 
requiring several new assessments for candidates pursuing certification as a teacher, coun- 
selor, librarian, principal, or superintendent. These assessments are designed to provide 
additional data for evaluating preparation programs and include the Missouri Stan- 
dards'Based Performance Assessments, which measure candidate performance in content, 
coursework, and clinical experiences; the Missouri Educator Profile, designed to identify 
and develop work habits associated with effective educators; the Content Specialty and 
Pedagogy Assessments, which are exit exams aligned with Missouri and national standards 
that will replace the Praxis Series tests; and the Missouri General Education Assessment, 
an admission test for undergraduate teacher preparation programs, which measures knowh 
edge in English/language arts, writing, math, science, and social sciences and will replace 
the College Basic Academic Subjects Examination currently in use. 

Nebraska. The state is developing a graduate follow-up survey and plans to develop 
common tools to assess the content knowledge and performance of teacher candidates. 


Colorado has 
made substantial 
investments to 
ensure that state 
education agency 
data systems 
are adequate to 
answer the types 
of evaluation 
questions the 
agency seeks 
to ask 


Enhancements to state data systems 

Colorado has made substantial investments to ensure that state education agency data 
systems are adequate to answer the types of evaluation questions the agency seeks to ask. 
In response to legislative changes related to evaluation, state agency staff recently worked 
with a consortium of researchers from Colorado universities to assess the feasibility of 
using student outcome data to evaluate teacher preparation programs by examining the 
state’s ability to link teachers to the programs that prepared them. This study focused on 
traditional programs, looking closely at the processes used to identify candidates who had 
completed their initial licensure and those who completed additional endorsements in 
high-need areas (Alzen et al., 2012a, 2012b). To ensure that teachers’ initial licensure and 
endorsements were attributed to the correct programs in the correct year and to confirm 
that teachers had completed the program, teacher rosters were sent for verification to 
teacher preparation programs. Data from this verification process were then used to assess 
the validity of a process for creating links between teachers and programs in cases where 
a program is unable to retroactively identify program graduates. The process used existing 
data and identified a substantial number of “false positives” (teachers who were incorrectly 
attributed to a preparation program), raising concern about its use. 

As a result of these findings, the Colorado Department of Higher Education now requires 
teacher preparation programs to identify graduates of initial licensure programs as part of 
their reporting to the state. The study also provided recommendations to better link teach- 
ers to their preparation programs, such as by increasing use of campus student information 
systems to track candidate data, using a common definition of graduates across programs, 
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including licensure exam scores in campus data systems, and developing internal processes 
to verify the accuracy of data before submission to the state. A state agency staff member 
reported that the state has invested substantial resources to ensure that data on teacher 
preparation programs are accurate. 

New approaches for reporting evaluation findings 

Results from program approval and reauthorization reviews are typically shared only with 
the professional education unit. Although results are sometimes available to the public 
on request, state agency staff indicated that the results are rarely requested. Colorado and 
Missouri are exploring approaches for more frequent and more public reporting. Nebras- 
ka is in the early stages of considering new approaches for public reporting of evaluation 
findings. 

Colorado. For roughly a decade Colorado state law has required an annual report to the 
state General Assembly that describes the state’s approved teacher preparation programs. 
Recent legislation requires annual reports on the effectiveness of educator preparation pro- 
grams, using data collected during an educator’s first three years of employment and that 
include information about student academic growth, educator placement, and educator 
mobility and retention to assess how programs perform. According to the law, the Colora- 
do Department of Education is expected to collaborate with teacher preparation programs 
and the Colorado Department of Higher Education to share reports of annual evaluations 
in order to improve curricula and programs. The reports are expected to be made available 
to the public on its website. 

Colorado state education agency staff reported that the state has yet to determine how 
results from future program evaluations will be made public and that they want to ensure 
that the reporting process encourages positive use of the results: “We have a commit- 
ment to make sure that the data are used in a positive way and not in a harmful way . . . 
[Programs] are very concerned. They lived through a previous time when there was this 
ranking of [teacher preparation programs] and we don’t want to use the information pre- 
maturely to bash a preparation program, especially when it’s not appropriate . . . When we 
get to the public reporting of summarizing the information, we want to make sure that the 
method in which we present the information is done carefully.” 

Missouri. State agency staff reported that a new format is being developed for annual 
performance reports that focuses on teacher candidate outcome measures. A pilot of the 
revised report format is expected to be implemented in 2014. Programs will have the oppor- 
tunity to review findings and appeal any findings of concern before they are made public. 
A state agency staff member mentioned that the state also intends to develop a system that 
teacher preparation programs can use to generate reports based on data the state collects. 

Nebraska. A state agency staff member indicated that public reporting of the evalua- 
tion findings for teacher preparation programs is under consideration and that the state 
is concerned about ensuring the validity of what is reported: “In terms of the Nebraska 
state report card, I say it might be a couple of years . . . it’s a concern about what do you 
report on a report card that fairly, reliably, and validly indicates the quality of an educator 
program.” 


Colorado and 
Missouri are 
exploring 
approaches for 
more frequent 
and more public 
reporting, and 
Nebraska is in the 
early stages of 
considering new 
approaches for 
public reporting of 
evaluation findings 
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Implications of study findings 


States are moving from program approval and reauthorization processes designed to ensure 
compliance with standards to more frequent evaluation activities that focus more closely 
on program outcomes for teacher candidates, practicing teachers, and their students. States 
are developing and implementing new data collection tools and making enhancements 
to data systems that will permit this focus on candidate, teacher, and student outcomes. 
States are also exploring ways to report evaluation results to support public accountability 
for outcomes and program improvement. These more frequent and more outcomeTocused 
approaches to evaluation have the potential to motivate a change from the current focus 
on program accountability to meaningful and ongoing identification of program strengths 
and weaknesses that can be used to improve programs. 

Common purposes for evaluating teacher preparation programs include holding programs 
accountable, providing consumer information to prospective teacher preparation program 
students and their potential employers, and supporting program improvement (Feuer et 
al., 2013). As states develop new evaluation systems, it will be important to keep these 
purposes at the center of discussions about evaluation system design and the best use of 
evaluation findings. 

Future research on teacher preparation program evaluation is needed to understand the 
utility of various evaluation approaches in achieving these different purposes. For example, 
to ensure that evaluations are meaningful, more information is needed on how state evalua- 
tions can best complement other evaluation activities promoted by the federal government 
(such as those required for Title II reporting) and organizations such as NCATE/CAEP 
and the National Council on Teacher Quality and those undertaken by programs them- 
selves. Research should also explore how to ensure the reliability and validity of evaluation 
data and the types of data and dissemination that best facilitate program improvement. 

Limitations of the study 


More frequent 
and more 
outcome-focused 
approaches to 
evaluation have 
the potential to 
motivate a change 
from the current 
focus on program 
accountability 
to meaningful 
and ongoing 
identification of 
program strengths 
and weaknesses 
that can be used to 
improve programs 


This report offers a state-level perspective on teacher preparation program evaluation, 
based on interviews with one informant and review of publicly available documents in 
each state. The availability of documentation varied across states, and data were collected 
at a single point in time. Continually evolving state plans, policies, and procedures may 
have been updated since the completion of data collection in October 2013. 

Despite these limitations, this report is one way to gauge state efforts and plans related 
to teacher preparation program evaluation at a time when state approaches are changing 
nationwide. 
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Appendix A. Data and methodology 


This appendix describes the study design, data collection, data analysis, and provisions to 
protect respondent confidentiality. 

Study design 

This descriptive study was conducted between May and October 2013 and included doc- 
ument review from all REL Central states (Colorado, Kansas, Missouri, Nebraska, North 
Dakota, South Dakota, and Wyoming) and interviews with state education agency repre- 
sentatives with primary responsibility for evaluating teacher preparation programs in six of 
the seven REL Central states (all but Wyoming). 

Publicly available documents were reviewed to collect information about current and 
planned approaches for evaluating teacher preparation programs, including documents 
that described data sources, methods, and findings related to evaluation. Because approach- 
es for program evaluation are undergoing change in REL Central states and across the 
country and because much of the work in this area is very recent, document review was 
limited to publications produced in 2007 or later. Types of documents reviewed included 
state documents describing current and planned policies or practices for program evalu- 
ation; state-vetted reports of evaluation methods and findings; articles in peer-reviewed 
journals and other publications that document data sources and methods used for eval- 
uation; and conference papers, presentations, master’s theses, and doctoral dissertations 
related to evaluation. 

Because many state approaches to evaluating teacher preparation programs are under 
development, limited documentation is available about emerging methods. Interviews 
gathered in-depth information from the perspective of a state education agency staff 
member directly involved in evaluating teacher preparation programs. A representative of 
each state education agency was identified as the person best able to respond to questions 
about the state’s current and planned evaluation efforts. 

Participants were recruited through email and phone contact with state education agency 
staff identified by the study team as either a potential respondent or someone who could 
identify the appropriate respondent. These individuals received information about the 
study and a copy of the interview protocol and were asked to identify the state education 
agency staff member best able to address the questions. In each case the representative 
initially identified by the study team was the respondent who participated in the interview. 
In two states the respondent referred selected questions to another state education agency 
staff member who provided additional information by email. Respondents were state edu- 
cation agency staff members with primary responsibility for oversight and approval of 
teacher preparation programs. 

Data collection 

Publicly available documents were accessed from websites, including those associated with 
state education agencies, departments of higher education, journals, conferences, colleges 
and universities, and research organizations. Keyword searches were conducted using state 
education agency websites for each REL Central state; electronic databases, including 
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the Education Resources Information Center, Academic Search Premier, and Education 
Research Complete; and archived conference programs for the American Educational 
Research Association, Society for Research on Educational Effectiveness, and national 
conferences hosted by the National Center for Education Statistics. Search terms included 
the following: (“Colorado” or “Kansas” or “Missouri” or “Nebraska” or “North Dakota” or 
“South Dakota” or “Wyoming”) AND (“teacher preparation” or “educator preparation” or 
“teacher education” or “teacher training”) AND (“achievement” or “outcomes” or “reten- 
tion” or “assessment” or “evaluation” or “accountability” or “effectiveness”). 

The review included any documents on current or planned efforts to evaluate teacher 
preparation programs in REL Central states produced in 2007 or later, including descrip- 
tions of current or planned efforts, data sources, analysis methods of those efforts, and doc- 
umentation of findings from those efforts. Documents that appeared to meet these criteria 
were collected and reviewed by the principal investigator and at least one other member of 
the study team who each made a recommendation about whether to include the document 
in the review. Consensus guided whether the document was included in the study. Of the 
137 documents identified and screened for review, 57 were excluded because they were 
not relevant to the research questions. The number of documents reviewed for each state 
ranged from 6 to 23. The 80 documents reviewed are listed by state in appendix B. 

Some documents, such as federal grant and Elementary and Secondary Education Act 
waiver applications, policy briefs, and news stories from media outlets, were treated as 
secondary data sources — reviewed to inform conversations with interview participants — 
because information may have been inaccurate or represented plans that were not imple- 
mented. Only documents published by state entities with oversight for teacher preparation 
program approval were treated as primary data sources. 

Interviews were conducted by telephone with the selected state education agency staff 
members in six of seven REL Central states. A respondent from Wyoming could not be 
reached to participate in an interview. Interview protocols were sent by email to respon- 
dents several days before the interview so that they could prepare for the interview by 
gathering relevant documents and consulting with colleagues. At least two members of 
the study team were present at each interview. Interviewers reviewed documentation to 
become familiar with state policies, initiatives, and language specific to the state’s program 
evaluation efforts and reviewed the interview protocol to ensure a common understanding 
of its questions and probes. Interviews lasted about an hour and were digitally recorded 
and transcribed. 

The interview protocol included questions intended to capture aspects of the state’s 
current and planned evaluation activities (see appendix C). After the first interview, the 
study team met to review the protocol and responses and concluded that the protocol was 
effective and that no changes were needed. 

Data analysis 

Analysis of information collected through document review and interviews focused on 
each of the following content domains, intended to reflect key elements of state evaluation 
activities: questions being asked or objectives being served by the program evaluation, stan- 
dards for teacher preparation programs, measures, reporting conventions, state education 
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agency use of evaluation findings, and confidentiality provisions. Two study team members, 
working closely with the principal investigator, reviewed the documents and coded the 
contents into the content domains. After completing the document coding, researchers 
wrote a document review summary for each state. Three researchers wrote summaries, 
with two writing them for two states each and one writing them for three states. The study 
team compared the document summaries to ensure a similar level of detail and to assess 
commonalities and variances. Study team members reviewed interview transcripts, coding 
text (sentences or paragraphs) by content domains. Additional categories were created to 
address topics that emerged in interviews, such as state legislation related to program evah 
uation and state efforts to support program evaluation by school districts and institutions 
of higher education. 

To ensure uniform understanding of the content domains, study team members inde- 
pendently coded documents and interview transcripts for one state and discussed coding 
decisions. The study team discussed and reached consensus on the few coding discrepan- 
cies identified, selecting the domains that provided the best fit for the text and clarifying 
definitions as needed. 

The study team generated reports of coded text from documents and interviews for each 
content domain that included all assigned text segments. These reports were used to create 
written summaries of findings from document review and interviews by state. Summaries 
were compared across states to identify similarities and differences, focusing on organiza- 
tion of information and level of detail. To ensure accuracy of the summaries, relevant draft 
report sections were sent to each state for review. Respondents from six states provided 
feedback, and minor changes were incorporated into the reports for four states. From these 
summaries, researchers created a matrix to identify similarities and differences across states 
in each domain, which was used as the basis for the report findings. 

Protection of confidentiality 

Respondents were told that information collected as part of the study would be treated 
confidentially and that no names would be used in any report. However, they were also 
told that, because of the small sample of respondents, a knowledgeable reader of the final 
report might be able to identify the individual from whom information was collected for a 
particular state. However, the information collected focuses on state agency practices and 
plans that would not be considered confidential or sensitive. 

Respondents provided verbal consent to participate in the study after being provided with 
written and verbal information about the potential use of information gathered for this 
report. To maximize confidentiality, respondent names were removed from transcripts and 
interview notes. The computers on which the study team saved transcripts and analysis 
documents were password protected and available only to the study team. Digital files of 
interview recordings were destroyed after transcription, and all documents and data will be 
destroyed three years after completion of this study. 
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Appendix B. List of documents reviewed 


Colorado 

Adams, S., & Wolf, K. (2008). Strengthening the preparation of early childhood teacher 
candidates through performance-based assessments. Journal of Early Childhood Teacher 
Education, 29(1), 6-29. 

Alzen, J., Briggs, D., Whitcomb, J., Haug, C. Paterson, W., & Klopfenstein, K. (2012). 
An initial exploration of Colorado'trained teachers: Providing context for outcomesTased 
teacher preparation program evaluation. University of Colorado, Boulder. 

Alzen, J., Briggs, D., Whitcomb, J., Haug, C. Paterson, W., & Klopfenstein, K. (2012). 
Enhancing Colorado data systems: Linking teachers to preparation programs. University of 
Colorado, Boulder. 

Colorado Commission on Higher Education. (2013). Report the General Assembly: Program 
results for educator preparation FY 2011—2012. Denver, CO: Author. 

Colorado Department of Education. (2009). Application for grants under the Statewide 
Longitudinal Data System Grant Program. Denver, CO: Author. 

Colorado Department of Education. (2010). Commissioner of Education Dwight D. Jones 
leads NCATE call for overhaul of teacher preparation programs [News release]. Retrieved 
October 23, 2013, from http://www.cde.state.co.us/communications/Releases/20101116 
ncate.html 

Colorado Department of Education. (2010). Race to the Top application: Round 1. Denver, 
CO: Author. 

Colorado Department of Education. (2010). Race to the Top application: Round 2. Denver, 
CO: Author. 

Colorado Department of Education. (2011). Race to the Top application: Round 3. Denver, 
CO: Author. 

Colorado Department of Education. (2012). Colorado’s application for a waiver from the 
requirements of NCLB parts 1 and 2. Denver, CO: Author. 

Colorado Department of Education, (n.d.). Educator preparation program approval and 
reauthorization. Retrieved October 23, 2013, from http://www.cde.state.co.us/cdeprof/ 
EPP_index.htm 

Colorado Department of Education, (n.d.). Preparation: Training educators who will 
make a difference. Retrieved October 23, 2013, from http://www.cde.state.co.us/ 
educatoreffectiveness/prepare 

Colorado General Assembly. (2010). Session laws of Colorado 2010: Second regular session, 
67th General Assembly. Denver, CO: Author. 
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Engdahl, T. (2010, November 16). Study: Turn teacher prep “upside down.” EdNews 
Colorado. Retrieved August 23, 2013, from http://www.ednewscolorado.org/news/ 
education'news/study'turn'teacher'prep'upside'down 

Greenberg, J., & Jacobs, S. (2009). Preparing tomorrow’s teachers: Are Colorado’s education 
school graduates ready to teach reading and mathematics in elementary classrooms 1 Wash- 
ington, DC: National Council on Teacher Quality. 

Macgillivray, I. (2011). Colorado’s legal and policy framework for using data to drive improve- 
ments in educator preparation [PowerPoint presentation]. Retrieved August 23, 2013, from 
http://www.nga.org/files/live/sites/NGA/files/pdf/1103PRINCIPALPREPMAC.PDF 

Sawchuk, S. (2013). Project seeks to trace development of novice teachers’ skills. Retrieved 
October 21, 2013, from http://blogs.edweek.org/edweek/teacherbeat/2013/03/project_ 
seeks_to_trace_develop.html 

Kansas 

Kansas State Department of Education. (2012). Regulations and standards for Kansas edu- 
cators. Topeka, KS: Author. Retrieved August 23, 2013, from http://www.ksde.org/ 
Portals/0/TLA/Licensure/Licensure%20Documents/CertHandbook8'2011%20FINAL. 
pdf 

Kansas State Department of Education. (2010). Race to the Top application: Round 1. 
Topeka, KS: Author. 

Kansas State Department of Education. (2011). Institutional handbook for program approval. 
Topeka, KS: Author. 

Kansas State Department of Education. (2011). Kansas Educator Evaluation Protocol 
(KEEP). Topeka, KS: Author. 

Kansas State Department of Education. (2012). Licensure standards revisions. Topeka, KS: 
Author. 

Kansas State Department of Education, (n.d.). ESEA flexibility (waiver) request. Topeka, 
KS: Author. 

Kansas State Department of Education, (n.d.). Kansas Performance Teaching Portfolio: 
KPTP 1 01 and candidate training. Topeka, KS: Author. 

Martin, K., & Coleman, P. (2011). Licensing teacher leaders: The Kansas model. Delta 
Kappa Gamma Bulletin, 77(3), 6-9. 

Teaching in Kansas Commission. (2009). Teaching in Kansas Commission Action Plan. 
Topeka, KS: Author. 
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Missouri 


Bock, J, (2012, December 26). Missouri raises the bar on teacher preparation. St. Louis 
Today. Retrieved August 23, 2013, from http://www.stltoday.com 

Ehlert, M., Koedel, C., Parsons, E., & Podgursky, M. (2012). Selecting growth measures for 
teacher evaluations. University of Missouri-Columbia. 

Ehlert, M., Podgursky, M., Harrington, J., Xiang, P, Parsons, E., Frazelle, S., & Taylor 
Knight, L. (2011). Teacher and leader preparation in relationship to student achievement in 
the Kansas City region. Kansas City Area Education Research Consortium. 

Elder, B. (2008). Missouri Department of Elementary and Secondary Education: First-year 
teacher survey, 2007 [PowerPoint presentation]. Retrieved August 23, 2013, from http:// 
oseda.missouri.edu/presentations/first_year_teacher_survey_2007_janll_2008.ppt 

Hairston, G., & MacCloud, H., (2012, November 20). Annual performance report: Educator 
preparation programs: Presentation to the state board of education [Video webinar]. Jeffer- 
son City, MO: Missouri Department of Elementary and Secondary Education. 

Kauffman Foundation. (2005). A study of teacher preparation in Kansas City. Kansas City, 
MO: Author. 

Kovar, K. (2012). Educator licensure best practices. Jefferson City, MO: Missouri Department 
of Elementary and Secondary Education. Retrieved August 23, 2013, from http://dese. 
mo.gov/topl0by20/documents/10x20'fmahreport'kovar'g3tla.pdf 

Missouri Advisory Council for the Certification of Educators. (2012, September 24). Biam 
nual council meeting [Meeting minutes]. Retrieved August 23, 2013, from http://dese. 
mo.gov/eq/cert/documents/September2012Notes.pdf 

Missouri Department of Elementary and Secondary Education. (2004). Missouri Standards 
for Teacher Education Programs (MoSTEP): Institutional handbook. Jefferson City, MO: 
Author. 

Missouri Department of Elementary and Secondary Education. (2006). Missouri Standards 
for Teacher Education Programs (MoSTEP) and benchmarks for preliminary teacher edu~ 
cation programs. Jefferson City, MO: Author. 

Missouri Department of Elementary and Secondary Education. (2007). Missouri Standards 
for Teacher Education Programs (MoSTEP): Examiner’s handbook. Jefferson City, MO: 
Author. 

Missouri Department of Elementary and Secondary Education. (2008). Application for 
grants under the Statewide Longitudinal Data System Grant Program. Jefferson City, MO: 
Author. 

Missouri Department of Elementary and Secondary Education. (2010). Race to the Top 
application: Round 1. Jefferson City, MO: Author. 
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Missouri Department of Elementary and Secondary Education. (2010). Race to the Top 
application: Round 2. Jefferson City, MO: Author. 

Missouri Department of Elementary and Secondary Education. (2012). ESEA flexibili- 
ty request. Jefferson City, MO: Author. Retrieved October 21, 2013, from http://dese. 
mo.gov/sites/default/files/qs'esea'waiver'mo'flexibility'request.pdf 

Missouri Department of Elementary and Secondary Education. (2012). Missouri Standards for 
the Preparation of Educators (M oSPE): List of program standards. Jefferson City, MO: Author. 

Missouri Department of Elementary and Secondary Education. (2012). Recruitment and 
retention of teachers in Missouri public schools: A report to the Missouri General Assembly. 
Jefferson City, MO: Author. 

Missouri Department of Elementary and Secondary Education. (2012). Results of the begin- 
ning teacher survey: Comparative trend summary 2007—2013. Jefferson City, MO: Author. 

Missouri Department of Elementary and Secondary Education. (2013, April 10). New 
assessments for future teachers being phased in over next two years [Press release]. 
Retrieved August 23, 2013, from http://dese.mo.gov/communications/newS'releases/ 
new-assessments'future'teachers'being'phased'over'next'two'years 

National Council for Accreditation for Teacher Education. (2010). Transforming teacher 
education through clinical practice: A national strategy to prepare effective teachers. Wash- 
ington, DC: Blue Ribbon Panel on Clinical Preparation and Partnerships for Improved 
Student Learning. 

Soden, G. J. (n.d.). A review of teacher education programs: What Missouri can learn from highly 
effective teacher education programs [Doctoral dissertation]. University of Missouri- 
Columbia. 

North Dakota 

Education Standards and Practices Board, (n.d.). Program approval: New performance-ori- 
ented teacher preparation standards adopted. Bismarck, ND: Author. Retrieved August 
23, 2013, from http://www.nd.gov/espb/progapproval/adopt.html 

North Dakota Department of Public Instruction. (2011). Superintendent of public instruc- 
tion: 2009-2011 biennial report. Bismarck, ND: Author. 

North Dakota Department of Public Instruction. (2012). North Dakota FY 2010-2011 
annual performance report for special education. Bismarck, ND: Author. 

North Dakota Department of Public Instruction, Teacher and Principal Evaluation Sub- 
committee. (2012). NDDP1 teacher and principal evaluation document 2: Principle 3 
guidelines outline. Bismarck, ND: Author. 

Olson, A. C. (2009). Teacher education admission requirements and student teacher evalua- 
tions: Relationships among grade point average, Praxis I scores, and student teacher final 
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World-Herald. Retrieved August 23, 2013, from http://www.omaha.com/article/ 
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Nebraska Department of Education. (2008). Regulations for the approval of teacher education 
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NE: Author. 
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Appendix C. Structured interview protocol 


The following introductory script and interview protocol were used when interviewing 
state education agency representatives. 

Introductory script 

Good morning (afternoon). My name is . I’m from RMC Research Corpo- 

ration, Denver, and I’m calling on behalf of REL Central — the federal Regional Education 
Lab that serves your state. We are conducting a study that is examining approaches for 
assessing teacher preparation programs in the seven REL Central states, including Colora- 
do, Kansas, Missouri, Nebraska, North Dakota, South Dakota, and Wyoming. Thanks for 
taking the time to participate. The purpose of this interview is to gather your knowledge 
about the approach that is used to assess your state’s teacher preparation programs and any 
plans your state may have for changes to that approach. This interview will take about 
forty to fifty minutes. 

Your participation in this study is voluntary and you may choose to stop participating at 
any time. There will be no penalty if you do not participate or choose to withdraw from 
the study. REL Central treats all data collected through interviews confidentially, and does 
not include names in any project report. The interview asks no questions of a personal 
or sensitive nature and does not solicit your opinion. Instead, it asks only for informa- 
tion about your state’s activities and plans. However, because we are interviewing a small 
number of respondents from each state and there are typically few respondents within a 
state department who are able to respond to questions on a topic as narrow as this, it is 
possible that you could be identified by readers of the report as an interview respondent for 
your state. 

Do you agree to participate in this study? 

We typically make audio recordings of interviews so that we can be sure to capture your 
responses verbatim, rather than relying on handwritten notes. Would that be okay with 
you? [IF “YES” BEGIN AUDIO RECORDING] Do you have any questions before we 
begin? 

Interview protocol 

1. Please describe your primary job responsibilities. PROBE: What is your role with 
respect to your state’s evaluation of teacher preparation programs? 

2. How long have you been in this position? 

The following questions are about your state’s approach for the assessment or evaluation 
of teacher preparation programs. We know that, in many states, these approaches are 
changing, so we are interested in hearing about any aspects of TPP evaluation that are 
currently in place as well as those that are undergoing changes or are planned. Also, as we 
ask questions, if documentation is available that would provide further information about 
these topics or other individuals who we might contact, please feel free to refer me to those 
sources. 
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3. How does your state define “teacher preparation program?” What programs are includ- 
ed or excluded? 

4- What are the primary analysis questions or objectives that underlie your state’s current 
efforts or plans to assess TPPs? 

5. Does your state have standards specifically related to the performance of teacher 
preparation programs? 

6. Are there changes underway or planned by your state related to teacher preparation 
program standards? 

Now we’d like to know about what measures are used by your state for the assessment of 

teacher preparation programs. 

7. What measures, if any, are used to assess TPP program completions, graduate job 
placement, and/or graduate persistence in teaching? Please describe. 

8. What is the source of these data? 

9. When and how often are they collected? 

10. For which programs and teachers are they available? 

11. What measures, if any, are used to evaluate the achievement or achievement growth of 
the students of practicing teachers for the purpose of TPP evaluation? Please describe. 

12. For which programs and teachers are they available? 

13. What measures, if any, are used to evaluate the classroom teaching of practicing teach- 
ers for the purpose of TPP evaluation? Please describe. 

14. What is the source of these data? 

15. When and how often are they collected? 

16. For which programs and teachers are they available? 

17. Are any survey data collected for the purpose of TPP evaluation, such as principal 
surveys that ask about the performance of beginning teachers or surveys of the gradu- 
ates of TPPs? Please describe. 

18. What is the source of these data? 

19. When and how often are they collected? 
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20. For which programs and teachers are they available? 

[IF MULTIPLE MEASURES ARE USED] How are multiple measures integrated for 
use in rating programs and how much weight is given to each? 

21. Are there changes underway or planned by your state related to measures for teacher 
preparation program assessment? 

22. What percentage of TPPs in your state are included in your state’s TPP evaluation 
efforts? 

[IF NOT 100%] Which programs are excluded and why? 

23. When making conclusions about TPP performance, does your state have minimum 
data requirements? For example, does the state require data for a minimum number of 
teachers on which to base conclusions about program performance? Please describe. 

24- How are practicing teachers linked to their teacher preparation program? 

25. Does your state use an overall indicator of TPP performance, such as an index or a 
categorical rating system such as “low,” “medium” or “high-performing?” [IF SO] Please 
describe the indicator and how that indicator is created. 

[IF THE STATE USES OR PLANS TO USE VALUE-ADDED ACHIEVEMENT 
DATA] Is there additional technical documentation that you can provide that pro- 
vides more information about how your state currently uses or plans to use value-added 
student achievement data to assess TPPs? For example, which variables are included in 
the model, which subject areas and grades are included in the assessment, and which 
metrics are used to report TPP performance? 

26. Are there changes underway or planned by your state related to sampling and analysis 
methods for teacher preparation program assessment? 

27. To whom and how are results from TPP evaluation activities reported? 

28. Does your state provide additional information to individual TPPs that is not present- 
ed in public reports? What data are presented in each type of report? 

29. Are there changes underway or planned by your state related to reporting results of 
teacher preparation program assessment? 

30. What accountability mechanisms, if any, are in place to address low performance 
among TPPs? 

31. Are there any provisions for recognizing high-performing TPPs? Please describe. 

32. In what other ways, if any, are TPP evaluation results used? 
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33. Are there changes underway or planned by your state related to how findings from 
TPP assessment will be used? 

34. What assurances are provided to teachers related to the confidentiality of data related 
to their effectiveness? 

35. What issues, if any, has your state faced with respect to sensitive teacher or teacher 
preparation program evaluation data? 

36. Are there any aspects of your state’s current or planned approach to TPP assessment 
that you haven’t mentioned? 

37. Do you have anything else that you would like to add that I haven’t asked about? 

Thank you so much for your time! 
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Notes 


1. This report focuses on teacher preparation rather than educator preparation, which 
includes formal preservice training for librarians, counselors, and school administra- 
tors as well as teachers because federal and state efforts to reform the way the prepara- 
tion programs are evaluated have focused on those that prepare teachers. 

2. Documents were reviewed in all states; however, an education official from Wyoming 
could not be reached to participate in the interview. 

3. For Title II reporting, states count multiple certification or licensure tracks at a single 
institution of higher education as a single program. 

4. For Title II reporting, program completers are those the state has documented as 
having met all the requirements of a state-approved teacher preparation program in 
a given year. Recommendation by the teacher preparation program to the state for 
an initial credential may or may not be used as a criterion for determining who is a 
program completer. 

5. For this purpose, the states define a “program” as each subject- or grade-level-spe- 
cific area in which an institution offers preparation for certification, licensure, or 
endorsement. 

6. In July 2013 the National Council for the Accreditation of Teacher Education and the 
Teacher Education Accreditation Council were consolidated into the Council for the 
Accreditation of Educator Preparation as a new national accreditation organization 
for educator preparation. Revised standards for accreditation of educator preparation 
were released in August 2013. See www.caepnet.org for more information. Because 
REL Central state agency staff still commonly refer to NCATE as the national accred- 
itation organization, the term “NCATE/CAEP” is used throughout this report. 

7. Educator preparation program approval in Colorado is based primarily on adherence 
to educator performance standards and additional endorsement standards reflected in 
the Colorado Educator Licensing Act of 1991 (§22-60.5 Colorado Revised Statute) 
and performance criteria expressed in Colorado Revised Statute §23-1-121(2) related 
to admission requirements, candidate advising and counseling, integration of theory 
and practice in coursework and field experience, supervised field based experience, and 
assessment of candidates’ content and pedagogical knowledge and skill. 

8. Rule 20 refers to the 2008 version of Chapter 20 of Title 92 of Nebraska Administra- 
tive Code. Under Rule 20 (Sections 005 through 007), all institutions must meet the 
basic requirements for program approval. Section 005 requires that programs have an 
officially designated administrator, admit only students who meet minimum scores on 
preprofessional assessments, inform students with certain criminal convictions that 
they are ineligible for certification, and provide required courses. Section 006 requires 
that programs have plans that identify teacher certification requirements, performance 
standards, and unit program standards and that programs provide assistance to new 
graduates. Section 007 requires institutions to provide a qualitative and quantitative 
report to the Nebraska Department of Education that provides institution- and pro- 
gram-level information about program details, plans, standards, and descriptors. 

9. In Colorado, Senate Bill 10-036, which requires the department of education to 
prepare an annual report analyzing educator preparation program effectiveness using 
data obtained through the educator identifier system, was passed in January 2010; 
Senate Bill 11-245, which concerns educator preparation programs at institutions of 
higher education, was passed in May 2011. 
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